WWC Intervention Report 


U.S. DEPARTMENT OF EDUCATION 


What Works Clearinghouse 


• 

• 1 ^ INSTITUTE OF 

■ EDUCATION SCIENCES 


Middle School Math 


March 26, 2007 



I CAN Learn® Pre-Algebra and Algebra 



Program description The / can Learn® Pre-Algebra and Algebra computerized cur- 
ricula are designed to cover mathematics and probiem-solving 
skiils for ethnicaily diverse, inner-city students in grades 6-12. 
The curricula are designed to equip students with the skills 
they need to meet district, state, and nationai math objec- 
tives through an interactive software program that teaches 



pre-aigebra and aigebra. The / CAN Learn® Pre-Algebra 
software program consists of 131 lessons, and Algebra of 181 
lessons. The developer describes the curriculum software 
as meeting National Council of Teachers of Mathematics 
standards and configurable to meet state and local grade-level 
expectations. 



Research 



One study of / CAN Learn® Pre-Algebra and Algebra met the 
What Works Clearinghouse (WWC) evidence standards and five 
more studies met WWC evidence standards with reservations. 
The six studies, which included more than 16,600 eighth- and 



ninth-grade students in middle and high schools in California, 
Florida, Georgia, and Louisiana, compared the standardized 
math performance of students who used the / CAN Learn® 
system with that of students who used traditional curricula.'' 



Effectiveness 



The / CAN Learn® Pre-Algebra and Algebra curricula were found to have positive effects on math achievement. 



Rating of effectiveness 
Improvement index^ 



Math achievement 

Positive effects 

Average: +6 percentile points 

Range: -7 to +20 percentile points 



1. The evidence presented in this report is based on the avaiiabie research. Findings and conciusions may change as new research becomes avaiiabie. 

2. These numbers show the average and range of improvement indices for aii findings across the studies. 



WWC Intervention Report I CAN Learn® Pre-Algebra and Algebra 



March 26, 2007 







Additional program 
information 



Research 



WWC Intervention Report 



Updating previous report 

This report updates the previous WWC report on / CAN Learn® 
that was released on the WWC website in November 2004. 

Since the reiease of the previous report, the WWC has updated 
its evidence standards and developed peer-review procedures 
for adjusting such methodologicai flaws in studies as mismatch 
between the unit of assignment and the unit of anaiysis and 
lack of adjustment for multiple comparisons. These standards 
and procedures, when applicable, have been applied to studies 
included the original / CAN Learn® Pre-Algebra and Algebra 
review. Six new studies were identified for this updated report. 

Developer and contact 

JRL Enterprises, Inc. Address: 400 Poydras Street, Suite 1000, 
New Orleans, LA 70130. Web: www.icanlearn.com. Email: sup- 
port@icanlearn.com. Telephone: (888) 263-1390. 

Scope of use 

The curriculum was first implemented in 1995. As of March 
2006, the / CAN Learn® system was being used in 16 elementary 
schools, 101 middle schools, and 123 junior and senior high 
schools across the United States. The / CAN Learn® system is 
typically used in large urban school districts and smaller rural 
school districts where students are predominantly at-risk and 
members of ethnic minority groups. 

Teaching 

The pre-algebra and algebra lessons include instructional vid- 
eos, interactive multimedia presentations, and a question bank. 
Both curricula were designed to connect math topics to “real 
world” applications. Lessons are grouped much like those in a 
textbook chapter. Students receive verbal and visual assistance 



in progressing through the lesson. After completing a lesson, 
students complete a cumulative review of the concepts taught, 
and teachers can monitor student progress through real-time 
assessment. 

Although much of the instruction is individualized for students 
in the / CAN Learn® system, teachers are expected to help 
determine the content of the lesson and other aspects of the 
class. At the beginning of the year the teacher determines the 
homework assignments, lesson organization, lesson presenta- 
tion, manipulatives, assessments, and grade evaluations. The 
program follows the teacher’s lesson plan and provides constant 
feedback to both the student and the teacher. According to the 
developer, teachers can accommodate different learning styles 
in a classroom of up to 30 students using the / CAN Learn® sys- 
tem. The / CAN Learn® system also aims to facilitate classroom 
management, since teachers can use the Classroom Explorer 
tool to record attendance, homework, and test grades and to 
chart individual student progress. 

As part of the / CAN Learn® system, teachers receive an initial 
two-day training in how to use the system. Teachers also have 
“best practices” training and unlimited access to training and 
on-site technical and pedagogical support in the classroom for 
three years. / CAN Learn® system personnel call or visit / CAN 
Learn® teachers weekly. 

Cost 

The cost of an / CAN Learn® classroom depends on its con- 
figuration and terms of support. A typical full installation— 30 
workstations in a classroom with all curriculum and classroom 
management software, computer hardware, network wiring, 
furniture, and three years of comprehensive onsite educational 
and technical support— costs $300,000, a one-time expense. 



In this updated review, 12 studies reviewed by the WWC 
investigated the effects of / CAN Learn® on students’ math 
achievement. One study (Kirby, 2006 October) was a randomized 
controlled trial that met WWC evidence standards. Two studies 



(Kerstyn, 2001; Kerstyn, 2002 October) used a quasi-experimen- 
tal design that met WWO standards with reservations. Three 
studies (Kirby 2004 September; Kirby, 2004a November; Kirby, 
2005 January) were randomized controlled trials that met WWO 
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evidence standards with reservations because of teacher-inter- 
vention confounding problems. The remaining six studies did not 
meet WWC evidence screens.^ Although each study investigated 
the effect of / CAN Learn® on a narrow population and setting, 
collectively the studies investigated and reported on middle 
school students of diverse racial and economic backgrounds In 
urban, suburban, and rural districts. 

Met evidence standards 

Kirby (2006, October) assessed the impact of the / CAN Learn® 
system on 2003-04 math achievement by randomly assigning 
students to intervention or comparison classrooms. The study 
included 2,400 eighth-grade regular education students from 13 
Orleans Parish Public Schools. / CAN Learn® classrooms were 
compared with classrooms using a traditional curriculum. 

Met evidence standards with reservations 

Kerstyn (2001) used a classroom matched-pairs quasi- 
experlmental design to Investigate the effect of the first year of 
implementation of the / CAN Learn® system on math achieve- 
ment of eighth-grade students in Hillsborough Oounty Public 
Schools in Tampa, Florida. At the beginning of the 2000-01 
school year, 58 / CAN Learn® classrooms (with 1,222 students) 
were matched with 58 traditional mathematics classrooms (with 
1,314 students). The / CAN Learn® system was implemented with 
four separate samples of students enrolled in four math courses: 
Algebra 1 (8 classes, 175 students). Algebra 1 Honors (8 classes, 
150 students), MJ-3 pre-algebra (32 classes, 678 students), and 
MJ-3 Advanced (10 classes, 219 students). The effectiveness of 
the / CAN Learn® system is reported for each of the four study 
samples in the Findings section. 



Kerstyn (2002, October) continued the investigation of the / 
CAN Learn® system during the second year of implementation in 
Hillsboro County Public Schools with a quasi-experimental study 
of a different sample of 11,125 eighth-grade students in a total 
of 597 classrooms. The / CAN Learn® system was implemented 
with four separate samples of students enrolled in four math 
courses: Algebra 1 (10 classes, 188 students). Algebra 1 Honors 
(10 classes, 188 students), MJ-3 pre-algebra (64 classes, 1,028 
students), and MJ-3 Advanced (37 classes, 424 students). The 
effectiveness of the / CAN Learn® system is reported for each of 
the four study samples in the Findings section. 

Kirby (2004, September) assessed the impact of the / CAN 
Learn® system on 2003-04 math achievement by randomly 
assigning 204 eighth-grade students either to one teacher using 
the / CAN Learn® mathematics curriculum or two teachers using 
a traditional math curriculum in Bret Harte Middle School in 
Alameda County, California. The comparison classrooms used 
the state-adopted Glencoe pre-algebra textbook. Because there 
was only one / CAN Learn® teacher, it is not possible to separate 
the effect of the teacher from the effect of the / CAN Learn® 
system. Thus, even though students were randomly assigned, 
this study met WWC standards with reservations. 

Kirby (2004a, November) randomly assigned 254 students 
either to / CAN Learn® classes or comparison classes. In the / 
CAN Learn® classes, one teacher facilitated instruction using the 
computerized curriculum. Students in the comparison classroom 
used a traditional math curriculum delivered by their teachers. 
Because there was only one / CAN Learn® teacher, it is not pos- 
sible to separate the effect of the teacher from the effect of the / 
CAN Learn® system. Thus, even though students were randomly 
assigned, this study met WWC standards with reservations. 



3. One study, Kirby (2004b, November), failed to meet evidence screens in the original review because of missing data for effect-size computations, but 
was reclassified as meets WWC standards in this updated report based on updated information (Kirby, 2006 October). A second study, Kirby (2004a, 
November), met WWC standards in the original review but met WWC standards with reservations in this updated review because only one teacher 
delivered the instruction using the / CAN Learn^ system, and the WWC had reservations about the confounding of teacher characteristics with the 
intervention. A third study (Kirby, 2005 January) that met evidence standards with reservations was a new study added to this report. 
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RSSSarch (continued) Kirby ( 2005 , January) randomly assigned 137 ninth-grade 

students either to / CAN Learn® classrooms or traditional math- 
ematics classrooms in a high school in Catoosa County, Geor- 
gia, in 2004. Because there was only one / CAN Learn® teacher, 



it is not possible to separate the effect of the teacher from the 
effect of the / CAN Learn® system. Thus, even though students 
were randomly assigned, this study met WWC standards with 
reservations. 



Effectiveness Findings 

The WWC review of middle school math addresses student 
outcomes in the math achievement domain. 

Kirby (2006, October) reported that the / CAN Learn® group 
statistically significantly outperformed the comparison group 
on the math exam from the Louisiana Educational Assessment 
Program test. The WWC confirmed this finding after correcting 
the statistical significance level for clustering. 

Kerstyn (2001) reported positive but not statistically significant 
effects of / CAN Learn® classrooms over comparison classrooms 
for Algebra 1, Algebra 1 Honors, MJ-3 pre-algebra, and MJ-3 
Advanced courses. The effect size for each of these samples 
was not large enough to be considered substantively important 
according to WWC criteria (at least 0.25). 

Kerstyn (2002, October) reported, and the WWC analysis 
confirmed, a positive and statistically significant effect for / CAN 
Learn® students in MJ-3 pre-algebra classes. The author also 
reported negative effects for students in the Algebra 1, Algebra 1 
Honors, and MJ-3 Advanced courses. But the WWC concluded 
that these negative effects were neither statistically significant 
nor large enough to be considered substantively important 
by WWC criteria. Thus, / CAN Learn® showed a statistically 
significant positive effect for MJ-3 pre-algebra students and 
indeterminate effects for Algebra 1, Algebra 1 Honors, and MJ-3 
Advanced students."^ 

Kirby (2004, September) reported that the / CAN Learn® group 
statistically significantly outperformed the comparison group on 
the General Mathematics exam from the California Standards 



Test. The statistical significance of this effect was confirmed by 
WWC analysis. 

Kirby (2004a, November) reported that the / CAN Learn® 
group statistically significantly outperformed the comparison 
group on the Math exam from the Georgia Criterion-Referenced 
Competency Test. The statistical significance of this effect was 
confirmed by WWC analysis. 

Kirby (2005, January) reported that a higher percentage of 
/ CAN Learn® students than comparison students passed the 
Algebra 1 End-of-Course Test in Georgia. The statistical signifi- 
cance of this effect was confirmed by WWC analysis. 

In sum, in the math achievement domain, the WWC reviewed 
findings from 12 samples reported in six studies.® Five of these 
samples showed statistically significant positive effects, and the 
remaining seven samples showed indeterminate effects. One of 
the samples was examined in a study that used a strong design. 

Rating of effectiveness 

The WWC rates the effects of an intervention in a given outcome 
domain as positive, potentially positive, mixed, no discernible 
effects, potentially negative, or negative. The rating of effective- 
ness takes into account four factors: the quality of the research 
design, the statistical significance of the findings,® the size of the 
difference between participants in the intervention and compari- 
son conditions, and the consistency in findings across studies 
(see the WWC Intervention Rating Scheme) . 



4 . Findings for subgroups, such as MJ-3 pre-algebra students who were not Florida Comprehensive Assessment Test (FCAT) exempt and MJ-3 pre- 
algebra students exempt from the FCAT, are reported in Appendix A4, but are not inciuded in the WWC rating of effectiveness for the intervention. 

5. The four courses in the Kerstyn (2001; 2002 October) studies— Aigebra 1, Aigebra 1 Honors, MJ-3 pre-aigebra, and MJ-3 Advanced— were treated as 
separate studies because they examined effects of / CAN Learn® on different sampies of students using different curricuia. 
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The WWC found / CAN Lean^ 
Pre-Algebra and Algebra 
to have positive effects 
on math achievement 



Improvement index 

The WWC computes an improvement index for each individual 
finding. In addition, within each outcome domain, the WWC 
computes an average improvement index for each study and 
an average improvement index across studies (see Technicai 
Details of WWC-Conducted Computations) . This improvement 
index represents the difference between the percentiie rank 
of the average student in the intervention condition versus 
the percentiie rank of the average student in the comparison 
condition. Uniike the rating of effectiveness, the improvement 
index is entireiy based on the size of the effect, regardiess of 
the statistical significance of the effect, the study design, or the 
anaiyses. The improvement index can take on values between 
-50 and +50, with positive numbers denoting results favorable to 
the intervention group. 



The average improvement index for math achievement is +6 
percentiie points across the six studies, with a range of -7 to +20 
percentiie points across findings. 

Summary 

The WWC reviewed 12 research reports on the / CAN Learn® 
Pre-Algebra and Algebra curricuia for this updated curricuium 
report. Of these, one study met WWC standards and five studies 
(which reported on 11 student samples or sub-studies) met 
WWC standards with reservations. The remaining six studies 
did not meet WWC evidence screens. Based on the findings 
reported in these studies, the WWC concluded that the / CAN 
Learn® system has a positive effect on math achievement. The 
evidence presented in this report may change as new research 
emerges. 



Refersnces Met WWC evidence standards 

Kirby, P. C. (2006, October). / CAN Learn® In Orleans Parish 
Public Schools effects on LEAP 8th grade math achievement, 
2003-2004. (Avaiiable from ed-cet, Inc., 2301 Killdeer Street, 
New Orleans, LA 70122) 

Additional source: 

Kirby, P. C. (2004b, November). / CAN Learn® in Orleans 
Parish Public Schools effects on LEAP 8th grade math 
achievement, 2003-2004. (Available from ed-cet, Inc., 2301 
Killdeer Street, New Orleans, LA 70122) 

Met WWC evidence standards with reservations 

Kerstyn, C. (2001). Evaluation of the I CAN Learn® mathematics 
classroom: First year of implementation (2000-2001 school 
year). (Available from the Division of Instruction, Hillsborough 
County Public Schools, 901 East Kennedy Blvd., Tampa, FL 
33602) 



Kerstyn, C. (2002, October). Evaluation of the I CAN Learn® 
mathematics classroom: Second year of implementation 
(2001-2002 school year). (Available from the Division of 
Instruction, Hillsborough County Public Schools, 901 East 
Kennedy Blvd., Tampa, FL 33602) 

Kirby, P. C. (2004, September). Comparison of I CAN Learn® and 
traditionally-taught 8th grade general math student perfor- 
mance on the California Standards Test, Spring 2004. (Avail- 
able from ed-cet, Inc., 2301 Killdeer Street, New Orleans, LA 
70122) 

Kirby, P. C. (2004a, November). Comparison of I CAN Learn® 
and traditionally-taught 8th grade student performance on the 
Georgia Criterion-Referenced Competency Test. Unpublished 
manuscript. 

Kirby, P. C. (2005, January). / CAN Learn® Algebra I in Catoosa 
County, Georgia. (Available from ed-cet, Inc., 2301 Killdeer 
Street, New Orleans, LA 70122) 



6. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within 

classrooms or schools and for multiple comparisons. For an explanation, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted 
Computations for the formulas the WWC used to calculate the statistical significance. In the case of / CAN Learn'^ Pre-Algebra and Algebra, corrections 
for clustering were needed. 
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References (continued) Did not meet WWC evidence screens 



Oescher, J. (2002, January). / CAN Learn® education success in 
Caiifornia. (Available from JRL Enterprises, Inc., 3520 General 
DeGaulle Drive, Suite 1100, New Orleans, LA 70114)® 



Brooks, C. (1999, August). Evaiuation of Jefferson Parish 

technoiogy grant: i Can Learn® Aigebra I. Unpublished report 
submitted to the Superintendent of Jefferson Parish Public 
Schools.^ 



Oescher, J., & Kirby, P. C. (2004, December). / CAN Learn® 
results in Dallas, Texas: 9th grade 2003-2004. (Available from 
JRL Enterprises, Inc., 3520 General DeGaulle Drive, Suite 
1100, New Orleans, LA 70114)® 



Brooks, 0. (2000, September). Evaluation of Jefferson Parish 
technology grant: I CAN Learn® Algebra I. (Available from the 
Department of Educational Leadership, University of New 
Orleans, New Orleans, LA 70148)® 



Scafide, K. (2004, November). Effects of I CAN Learn® on math 
achievement in Gwinnett County Middle School. (Available 
from JRL Enterprises, Inc., 3520 General DeGaulle Drive, 



Gill, J. 0., & Gifford, 0. S. (2001). Evaluation of Jefferson Parish 



technology grant: I CAN Learn® Algebra I. Unpublished manu- Suite 1100, New Orleans, LA 70114)® 
script. University of New Orleans, LA.® 



For more information about specific studies and WWC caicuiations, piease see the WWC f CAN Learn ® 
Pre-Alaebra and Algebra Technicai Appendices . 



7. Complete data were not reported: the WWC could not compute effect sizes. 

8. Lack of evidence for baseline equivalence: the study, which used a quasl-experimental design, did not establish that the comparison group was equiva- 
lent to the intervention group at baseline. 

9. Does not use a strong causal design: there was only one teacher in each study condition for this quasi-experiment, so the analysis could not separate 
the effects of the intervention from the effects of the teacher. 
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Appendix 



Appendix A1.1 Study characteristics: Kirby, 2006, October (randomized controiied triai) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2006, October). 1 CAN Learn® in Orleans Parish Pubiic Schoois effects on LEAP 8th grade math achievement, 2003-2004. (Availabie from ed-cet, inc. 2301 
Kiiideer Street, New Orleans, LA 70122.) 

Additional source: 

Kirby, P. 0. (2004b, November), i CAN Learn® in Orleans Parish Pubiic Schoois effects on LEAP 8th grade math achievement, 2003-2004. (Available from ed-cet, Inc., 
2301 Kiiideer Street, New Orleans, LA 70122.) 


Participants 


Participants were 2,400 eighth-grade students (1,082 / CAN Leanf’ students and 1,318 traditional mathematics students) from 57 math classes in 13 Orleans Parish Public 
Schools.^ Students were randomly assigned to intervention and comparison classrooms using the SASI Basic Scheduling software. Teachers were not randomized to condi- 
tions. Only students with no special education classification were included in the analyses. About 96% of the students in the sample were African-American, less than 1% 
were Elispanic, and less than 0.5% were Caucasian. ^ 


Setting 


The study took place in the Orleans Parish public school system, which includes the city of New Orleans. The participating schools were selected for this study based on two 
criteria: the schools included both traditional and / CANLeanP eighth-grade classes and those classes included at least 20 students each. 


Intervention 


Students were taught using the / CAN LearrP mathematics curriculum. The amount of time students used the / CAN Learrf’ system varied by school, with some students using 
only the test prep module and others completing up to 95 lessons. The average number of lessons completed was 12.1. 


Comparison 


Comparison students were taught in traditional classes with the teacher serving as the primary deliverer of instruction. The author does not provide further information on the 
curriculum. 


Primary outcomes 
and measurement 


The primary outcome measure was the Louisiana Educational Assessment Program (LEAP) Grade 8 Mathematics Exam. (See Appendix A2 for more detailed descriptions of 
outcome measures.) 


Teacher training 


Information on teacher training was not provided. 



1. The original sample included 14 schools. However, random assignment occurred in only 13 schools. For the purposes of this review, the study design and findings are reported for the subsample 
of 13 schools. Further, the study author conducted analyses of subsamples of students in addition to the total sample of 2,400 students in 13 schools. Those analyses focused on students of 
teachers who taught both conditions and students with matched pretest and posttest scores. For rating purposes, only the analysis of the total sample in the 1 3 schools was reviewed because, 
under the current WWC guidance, there is no attrition or confounding problem that might warrant a review of a subsample. In addition, the two additional analyses did not meet WWC evidence 
screens because of insufficient information on sample size and pretest equivalence. 

2. The study notes that these data were collected in the Orleans Parish Public Schools before Hurricane Katrina. The demographics in the school district have changed since the study was 
conducted. 
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Appendix A1.2 Study characteristics: Kerstyn, 2001 (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Kerstyn, C. (2001). Evaluation of the 1 CAN Learn® mathematics classroom: First year of impiementation (2000-2001 school year). (Available from the Division of Instruction, 
Hillsborough County Public Schools, 901 East Kennedy Blvd., Tampa, FL 33602.) 


Participants 


Participants were 2,539 eighth-grade students in 116 classrooms (58 / CAN Leanf’ classrooms with 1,225 students and 58 traditional classrooms with 1,314 students) in 
Title 1 middle schools. The study was limited to regular education students. The students were racially diverse and many were eligible for free or reduced-price lunches. The 
58 / CAN Leanf classrooms and 58 fradifional classrooms were used in fhe analysis. 


Setting 


The participating students were from middle schools in the Hillsborough County Public School system in Florida. This county includes the Tampa metro area. 


Intervention 


Students were taught using the / CAN Leanf mathematics curriculum. The author does not indicate how many of the lessons are required to be completed for the curriculum 
to be implemented as intended. The author indicates that the / CAN Leanf system was implemented in 45-, 50-, 80-, and 90-minute class periods. When surveyed, the 
teachers reported that 45 minutes was not long enough to make it through the curriculum. The intervention and evaluation occurred during the 2000-01 academic year. 


Comparison 


Students in classrooms that were selected as a match with intervention classrooms were taught in traditional classes using a traditional math curriculum. The author does not 
provide further information on the curriculum. 


Primary outcomes 
and measurement 


The primary outcome measure was the Florida Comprehensive Assessment Test (FCAT) Grade 8 Math Test.^ (See Appendix A2 for more detailed descriptions of outcome 
measures.) 


Teacher training 


Teachers in this study participated in training sessions on the use of the software and hardware, but not on use of the software in instruction. 



1 . A Semester 1 exam was also administered to study participants at the end of the first semester. The results are not reported in Appendix A3 with the results of the FCAT, because the psychomet- 
ric properties of the Semester 1 exam were not reported and it measured the effect of the intervention at an earlier point than did the FCAT. Because the FCAT measured math achievement at the 
end of the school year, the WWC reasoned that the FCAT was a more appropriate and valid measure of the intervention’s end-of-year effects. 
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Appendix A1.3 Study characteristics: Kerstyn, 2002, October (quasi-experimentai design) 



Characteristic 


Description 


Study citation 


Kerstyn, C. (2002, October). Evaluation of the 1 CAN Learn® mathematics classroom: Second year of Implementation (2001-2002 school year). (Available from fhe Division of 
Instruction, Hillsborough County Public Schools, 901 East Kennedy Blvd., Tampa, EL 33602.) 


Participants 


Participants were 11,125 eighth-grade students in 597 classes (129 / CANLeanf classes with 1,871 students and 468 traditional mathematics classes with 9,254 students) 
enrolled in Algebra 1, Algebra 1 Honors, MJ-3 pre-algebra, or MJ-3 Advanced math classes in the 41 middle schools in Hillsborough County Schools, Elorida. The analyses 
incorporate all students. Subgroup analyses present separate results for standard curriculum students and ECAT-exempt students. Compared with traditional classrooms, a 
higher proportion of / CAN LearrP students were on free and reduced-price lunch and were from minority backgrounds (African-American and Hispanic). 


Setting 


Participating students were from middle schools in the Hillsborough County Public School system in Elorida. This county includes the Tampa metro area. 


Intervention 


Students were taught using the / CAN LearrP Algebra course, which consists of 109 complete algebra lessons. The / CAN LearrP system is intended to be the primary source 
of instruction. The intervention and evaluation occurred during the 2001-02 academic year. 


Comparison 


Comparison students were taught using a traditional math curriculum. The author does not provide further information on the curriculum. 


Primary outcomes 
and measurement 


The primary outcome measure was the FCAT Grade 8 Math Test. (See Appendix A2 for more detailed descriptions of outcome measures.) 


Teacher training 


Information on teacher training was not provided. 
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Appendix A1.4 Study characteristics: Kirby, 2004, September (randomized controiied triai with teacher-intervention confound probiem) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2004, September). Comparison of 1 CAN Learn® and traditionaiiy-taught 8th grade general math student performance on the California Standards Test, Spring 
2004. (Available trom ed-cet, Inc., 2301 Killdeer Street, New Orleans, LA 70122.) 


Participants 


Participants were 204 eighth-grade students (91 / CAN Learif’ students and 113 traditional mathematics students) in Bret Harte Middle School. The / CAN LearrP classrooms 
contained a higher proportion of African-American students and a lower proportion of Hispanic and non-native English speaking students than the comparison classrooms. 


Setting 


The participating students were from Bret Harte Middle School, which is one of five middle schools in Hayward Unified School District in Alameda County, California. Hayward, 
southeast of San Francisco and south of Oakland, had a population of 144,633 and a mean household income of $51,177, according to the 2000 Census. The city is ethni- 
cally diverse, with Hispanics (34%), whites (29%), Asians (21%), and African-Americans (11%). 


Intervention 


Students were taught eighth-grade mathematics by one teacher using the / CANLearrf mathematics curriculum. The / CAN LearrP system consists of 303 lessons from basic 
mathematics to advanced algebra concepts. Teachers choose the lessons that align to local curricular goals. The / CAN LearrP system is intended to be the primary source of 
instruction. The intervention and evaluation occurred during the 2003-04 academic year. 


Comparison 


Comparison students were taught in traditional classes, with the teacher as the primary deliverer of instruction, using a curriculum based on the state-adopted Glencoe pre- 
algebra textbook. 


Primary outcomes 
and measurement 


The primary outcome measure was the General Mathematics exam from the California Standards Test (CST). (See Appendix A2 for more detailed descriptions of outcome 
measures.) 


Teacher training 


Information on teacher training was not provided. 
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Appendix A1.5 Study characteristics: Kirby, 2004a, November (randomized controiied triai with teacher-intervention confound probiem) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2004a, November). Comparison of i CAN Learn® and traditionally-taught 8th grade student performance on the Georgia Criterion-Referenced Competency Test. 
Unpublished manuscript. 


Participants 


Participants were 254 eighth-grade students (93 in / CANLeanf' classes and 161 in traditional classes). All participants were regular education students. 


Setting 


One middle school in the Gilmer County School District in northwestern Georgia. 


Intervention 


Students were taught by one teacher using the / CAN LearrP system. The intervention was implemented during the 2003-04 school year. 


Comparison 


Comparison students were taught in traditional classes, with the teacher serving as the primary deliverer of instruction. The author did not specify what curriculum was used. 


Primary outcomes 
and measurement 


The primary outcome measure was the math exam from the Georgia Criterion-Referenced Competency Test (GCRCT). (See Appendix A2 for more detailed descriptions of 
outcome measures.) 


Teacher training 


Only one / CAN LearrP teacher in this study was trained in the year prior to implementation. 
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Appendix A1.6 Study characteristics: Kirby, 2005, January (randomized controiied triai with teacher-intervention confound probiem) 



Characteristic 


Description 


Study citation 


Kirby, P. C. (2005, January). 1 CAN Learn® Algebra 1 in Catoosa County, Georgia. (Available trom ed-cet, Inc., 2301 Killdeer Street, New Orleans, LA 70122.) 


Participants 


Participants were 137 ninth-grade students (84 / CAN Learif' students and 53 traditional mathematics students) enrolled in Algebra 1 during the 2004 winter semester. 


Setting 


The participating students were trom suburban high schools (Lakeview-Fort Ogelthorpe Fligh School in Catoosa County, Georgia, and in the metropolitan area of Chattanooga, 
Tennessee). The student population is 95% white; 27% of the students were eligible for free or reduced-price school lunches. 


Intervention 


Students were taught Algebra 1 by one teacher using the / Can Team® A/gete curriculum. The / CANLeanf’ Algebra course consists of 177 lessons and a 23,550-question 
question bank in 17 units. The / CANLeanf system is intended to be the primary source of instruction. 


Comparison 


Comparison students were taught in traditional classes, with the teacher serving as the primary deliverer of instruction. Their curriculum was based on the Georgia Quality 
Core Curriculum standards, using the Larson, Boswell, Kanold, and Stiff (2001) Algebra /textbook published by McDougal Littell. 


Primary outcomes 
and measurement 


The primary outcome measure was the Georgia Algebra 1 End-of-Course Test (EOCT). 


Teacher training 


/ CAN Learrf classes were taught by a teacher trained in a two-day summer workshop on how to use the software and how to manage the instructional environment. 
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Appendix A2 Outcome measures in the math achievement domain 



Outcome measure 


Description 


Florida Comprehensive 
Assessment Test (FCAT) 
Grade 8 Math Test 


The FCAT math test is a standardized measure that includes items related to all five content strands of Florida’s Sunshine State Standards tor mathematics: number sense, 
concepts, and operations; measurement; geometry and spatial sense; algebraic thinking; and data analysis and probability (as cited in Kerstyn, 2001; Kerstyn, 2002 October). 
Test content at grade 8 is evenly divided among these five content strands. Students are given 160 minutes to take the exam, which includes multiple-choice items, gridded- 
response items, and performance tasks. Test results are reported as scale scores, which range from 100 to 500. 


General Mathematics 
exam from the California 
Standards Test (CST) 


The General Mathematics CST for grade 8 is based on the California Mathematics Standards for grades 6 and 7 (as cited in Kirby, 2004 September). The CST is administered 
to students in grades 8 and 9 who have not yet completed or are not enrolled in discipline-specific standards-based math courses, or who are enrolled in the first year of a 
multi-year Algebra 1 course. The CST consists of 65 multiple-choice questions. Test results are reported as scale scores, which range from 150 to 600. 


Louisiana Educational 
Assessment Program (LEAP) 
Grade 8 Mathematics Exam 


The LEAP Mathematics test for grade 8 includes a multiple-choice section and an open-ended section for more complex tasks (as cited in Kirby, 2006 October). The test is 
aligned to the state's Comprehensive Curriculum and Grade Level Expectations. Six strands of the Louisiana Mathematics Framework are represented in LEAP: number and 
number relations; algebra; measurement; geometry; data analysis, probability, and discrete math; and patterns, relations, and functions. Test results are reported as scale 
scores, which range from 100 to 500. 


Georgia Algebra 1 End- 
of-Course Test (EOCT) 


The state-mandated Algebra 1 EOCT is aligned with the Georgia Quality Core Curriculum standards (as cited in Kirby, 2005 January). As of Eall 2004, the EOCT score is fac- 
tored into students’ Algebra 1 course grade. The Algebra 1 EOCT contains 90 questions related to five content domains: algebraic fundamentals, operations on real numbers 
and algebraic expressions, solving equations and inequalities, functions and their graphs, and connections and applications. Students pass the EOCT by scoring 600 or above, 
and the figures reported are based on the dichotomous pass/fail measure. The study presents the proportion of students passing the test. 


Georgia Criterion- 
Referenced Competency 
Test (GCRCT) Math Test 


The GCRCT is designed to measure how well students acquire the skills and knowledge described in the Georgia Performance Standards and the Quality Core Curriculum (as 
cited in Kirby, 2004a November). The GCRCT Math Test contains 60 items in six areas: number sense and numeration; geometry and measurement; patterns, relationships, 
and algebra; statistics and probability; computation and estimation; and problem solving. Test results are reported as scale scores, which range from 150 to 450. 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


/ CAN l£anf‘ Comparison 

group group 


Mean difference^ 
(ICANLeanf‘- 
comparison) 


Statistical 
significance^ 
Effect size^ (at a = 0.05) 


Improvement 

index^ 



Kirby, 2006, October (randomized controlled trialf 



LEAP Math scale scores Grades 13/2,400 


295.30 

(37.60) 


278.82 

(43.50) 


16.48 


0.35 


Statistically 

significant 


+14 


Average^ for math achievement (Kirby, 2006 October) 








0.35 


Statistically 

significant 


+14 



Kerstyn, 2001, Algebra 1 (quasi-experimental design)^ 



FCAT mathematics Grades 16/350 


351.38 

(30.80) 


344.60 

(28.36) 


6.92 


0.23 


ns 


+9 


Average^ for math achievement (Kerstyn, 2001 Algebra 1) 








0.23 


ns 


+9 



Kerstyn, 2001, Algebra 1 Honors (quasi-experimental design)^ 



FCAT mathematics Grades 16/336 


372.99 

(34.47) 


373.73 

(35.80) 


-0.74 


-0.02 


ns 


-1 


Average^ for math achievement (Kerstyn, 2001 Algebra 1 Honors) 








-0.02 


ns 


-1 



Kerstyn, 2001, MJ-3 pre-algebra (quasi-experimental design)^ 



FCAT mathematics Grades 64/1,420 


296.77 

(31.19) 


293.89 

(38.09) 


2.88 


0.08 


ns 


+3 


Average^ for math achievement (Kerstyn, 2001 MJ-3 pre-algebra) 








0.08 


ns 


+3 



Kerstyn, 2001, MJ-3 Advanced (quasi-experimental design)^ 



FCAT mathematics Grade 8 20/430 


332.51 

(31.19) 


327.40 

(29.60) 


5.11 


0.17 


ns 


+7 


Average^ for math achievement (Kerstyn, 2001 MJ-3 Advanced) 








0.17 


ns 


+7 



Kerstyn, 2002, October, Algebra 1 (quasi-experimental design)^ 



FCAT mathematics Grades 66/1,195 


347.40 

(20.90) 


349.80 

(21.00) 


-2.36 


-0.11 


ns 


-5 


Average^ for math achievement (Kerstyn, 2002 October, Algebra 1) 








-0.11 


ns 


-5 



(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain (continued) 



Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


Author’s findings from the study 

Mean outcome 
(standard deviation^) 




WWC calculations 




ICANLearif^ 

group 


Comparison 

group 


Mean difference^ 
[1 CAN Learif^ - 
comparison) 


Effect size^ 


Statistical 
significance^ 
(at a = 0.05) 


Improvement 

index^ 








Kerstyn, 2002, October, Algebra 1 Honors (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


91/1,894 


369.60 


374.30 


-4.73 


-0.17 


ns 


-7 








(27.80) 


(27.60) 










Average^ for math achievement (Kerstyn, 2002 October, Algebra 1 Honors) 






-0.17 


ns 


-7 






Kerstyn, 2002, October, MJ-3 pre-algebra (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


328/5,957 


293.80 


289.90 


3.91 


0.10 


Statistically 


+4 








(32.00) 


(40.50) 






significant 




Average^ for math achievement (Kerstyn, 2002 October, MJ-3 pre-algebra) 






0.10 


Statistically 


+4 
















significant 








Kerstyn, 2002, October, MJ-3 Advanced (quasi-experimental design)^ 








FCAT mathematics 


Grade 8 


112/2,079 


329.90 


331.00 


-1.11 


-0.05 


ns 


-2 








(23.90) 


(24.10) 










Average^ for math achievement (Kerstyn, 2002 October, MJ-3 Advanced) 






-0.05 


ns 


-2 




Kirby, 2004, September (randomized controlled trial with teacher-intervention confound problem)^ 






Generai Mathematics CST 


Grade 8 


1/204 


315.60 


299.70 


15.85 


0.33 


Statistically 


+13 








(45.80) 


(49.70) 






significant 




Average^ for math achievement (Kirby, 2004 September) 








0.33 


Statistically 


+13 
















significant 






Kirby, 2004a, November (randomized controlled trial with teacher-intervention confound problem)^ 






GCRCT9 


Grade 8 


1/254 


333.50 


319.90 


13.60 


0.41 


Statistically 


+16 








(35.70) 


(31.70) 






significant 




Average^ for math achievement (Kirby, 2004a November) 








0.41 


Statistically 


+16 



significant 



(continued) 
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Appendix A3 Summary of study findings inciuded in the rating for the math achievement domain (continued) 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


/ CAN l£anf‘ Comparison 

group group 


Mean difference^ 
(ICANLeanf‘- 
comparison) 


Statistical 
significance^ 
Effect size^ (at a = 0.05) 


Improvement 

index^ 



Kirby, 2005, January (randomized controlled trial with teacher-intervention confound problem)^ 



Algebra 1 EOC test Grade 9 1/137 


0.86 

(0.35) 


0.72 

(0.45) 


0.14 


0.52 


Statistically 

significant 


+20 


Average^ for math achievement (Kirby, 2005 January) 








0.52 


Statistically 

significant 


+20 


Domain average^ for math achievement across all studies 








0.15 


na 


+6 



ns = not statistically significant 

na = not applicable 

1 . This appendix reports findings considered for fhe effecfiveness rafing and the average improvement indices. Subgroup findings from the same studies are not included in these ratings but are reported in Appendix A4. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. The mean difference is adjusted for pretest differences in Kerstyn (2002, October). For each sample 
reported in Kerstyn (2002, October), the control group mean is the intercept from the HLM model and the intervention group mean is this intercept plus the type of classroom (intervention or comparison) HLM beta coefficient. 

4. For an explanation of the effect size calculation, see iechnical Details of WWC-Conducted Computations . The effect size for Kirby (2005, January) is estimated based on the Cox Transformation (logs odds ratio divided by 1 .65). 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools and for multiple comparisons. For an explanation about the 
clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate statistical significance. In the case of Kerstyn (2001) and Kirby (2006), cor- 
rections for clustering were needed. No other studies required corrections for clustering or multiple comparisons. 

8. The WWC-computed average effect sizes for each study and for the domain across studies are simple averages rounded to two decimal places. The average improvement indices are calculated from the average effect size. 

9. The author reported results from the Georgia Criterion-Referenced Competency Test as scale scores and as criterion scores (that is, the percentage that passed the criterion score compared with the percentage that failed), but the 
WWC focused on the results from the scale scores because they contain more information than categorical scores. 
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Appendix A4 Summary of subgroup findings for the math achievement domain^ 









Author’s findings from the study 














Mean outcome 
(standard deviation^) 




WWC calculations 




Outcome measure 


Study 

sample 


Sample size 
(schools/ 
students) 


/ CAN l£anf‘ Comparison 

group group 


Mean difference^ 
(! CAN Learif‘ - 
comparison) 


Statistical 
significance^ 
Effect size^ (at a = 0.05) 


Improvement 

index^ 



FCAT mathematics 


Grade 8 


Kerstyn, 2002, October, MJ-3 pre-algebra standard curriculum (quasi-experimental design)^ 

258/4,045 294.20 295.10 -0.90 -0.03 

(33.90) (33.50) 


ns 


-1 


FCAT mathematics 


Grade 8 


Kerstyn, 2002, October, MJ-3 pre-algebra FCAT-exempt (quasi-experimental design)^ 

249/888 299.50 284.40 15.15 0.29 


Statistically 


+11 






(37.60) (52.90) 


significant 





ns = not statistically significant 

1 . This appendix presents additional subgroup findings for measures that fall in the math achievement domain. The findings used for rating purposes are presented in Appendix A3. 

2. The standard deviation across all students in each group shows how dispersed the participants’ outcomes are: a smaller standard deviation on a given measure would indicate that participants had more similar outcomes. 

3. Positive differences and effect sizes favor the intervention group; negative differences and effect sizes favor the comparison group. 

4. For an explanation of the effect size calculation, see Technical Details of WWC-Conducted Computations . 

5. Statistical significance is the probability that the difference between groups is a result of chance rather than a real difference between the groups. 

6. The improvement index represents the difference between the percentile rank of the average student in the intervention condition and that of the average student in the comparison condition. The improvement index can take on values 
between -50 and +50, with positive numbers denoting results favorable to the intervention group. 

7. The level of statistical significance was reported by the study authors or, where necessary, calculated by the WWC to correct for clustering within classrooms or schools (corrections for multiple comparisons were not done for findings 
not included in the overall intervention rating). For an explanation about the clustering correction, see the WWC Tutorial on Mismatch . See Technical Details of WWC-Conducted Computations for the formulas the WWC used to calculate 
statistical significance. In the case of Kerstyn (2002, October), no correction for clustering was needed. 
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Appendix A5 / Can Leanfi rating for the math achievement domain 



The WWC rates an intervention’s effects in a given outcome domain as positive, potentiaiiy positive, mixed, no discernible effects, potentially negative, or negative.'' 

For the outcome domain of math achievement, the WWC rated / CAN Learn® Pre-Algebra and Algebra as having positive effects. The remaining ratings (potentially 
positive effects, mixed effects, no discernibie effects, potentiaiiy negative effects, and negative effects) were not considered because / CAN Learn® was assigned the 
highest applicable rating. 



Rating received 

Positive effects: Strong evidence of a positive effect with no overriding contrary evidence. 

• Criterion 1: Two or more studies showing statisticaily significant positive effects, at ieast one of which met WWC evidence standards for a strong design. 

Met. Five studies of / CAN Learn® showed statistically significant positive effects. Of those, one study had a strong design according to WWC 
standards. 

• Criterion 1: No studies showing statisticaiiy significant or substantively important negative effects. 

Met. Five studies of / CAN Learn® showed statisticaiiy significant positive effects. The remaining seven studies showed indeterminate effects. No 
studies showed statisticaiiy significant or substantiveiy important negative effects. 

1. For rating purposes, the WWC considers the statistical significance of individual outcomes and the domain level effect. The WWC also considers the size of the domain level effect for ratings of 
potentially positive or potentially negative effects. See the WWC Intervention Rating Scheme for a complete description. 
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